Bandwidth-Efficient Collective Communication for Clustered Wide Area Systems

Metacomputing infrastructures couple multiple clusters (or MPPs) via wide-area networks and thus allow parallel programs to run on geographically distributed resources. A major problem in programming such wide-area parallel applications is the difference in communication costs inside and between clusters. Latency and bandwidth of WANs often are orders of magnitude worse than those of local networks. Our MagPIe library eases wide-area parallel programming by providing an efficient implementation of MPI’s collective communication operations. MagPIe exploits the hierarchical structure of clustered wide-area systems and minimizes the communication overhead over the WAN links. In this paper, we present improved algorithms for collective communication that achieve shorter completion times by simultaneously using the aggregate bandwidth of the available wide-area links. Our new algorithms split messages into multiple segments that are sent in parallel over different WAN links, thus resulting in a better utilization of the (scarce) wide-area bandwidth. To determine the optimal segment size and the best communication tree shape, we introduce a new performance model that extends LogP. We present communication algorithms for several collective operations, which we implemented in the MagPIe library. An experimental performance evaluation shows that the new algorithms significantly improve the performance of several collective operations (for large messages) and that there is a close match between the theoretical model and the measured completion times for most operations. Also, we show that our approach finds segment sizes and tree shapes that are close to optimal in most cases.